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. ABSTRACT: 

Apparatus for routing packets in a communication network comprises a plurality (100) of 
per-connection queues (20a--20i), each queue established for receiving packets from a 
respective source and temporarily storing received packets before routing to a particular 
destination; a weighted fair-queuing scheduler (40) for servicing packets from each of the plurality 
of per-connection queues at guaranteed pre-allocated rates; a sensing device for sensing a 
presence or absence of packets in queues, the absence of packets in queues indicating 
availability of excess bandwidth; and, a state dependent scheduler (200) for redistributing 
excess bandwidth upon sensing of queues absent packets, the state dependent scheduler (200) 
sen/icing those queues in accordance with a state variable corresponding to a performance 
property of the queues, wherein delay and isolation properties for routing packets of respective 
queues in weighted fair^iueutng is preserved. 



Best Available 




» 



(19) 




(12) 



(43) Date of pubiication: 

19.08.1998 Bulletin 1998/34 

(21) Application number: 98300754.3 

(22) Date off fning: 03.02.1998 



Europaisches Patentamt 
European Patent Office 
Office europ6endes brevets (11) EP 0 859 492 A2 

EUROPEAN PATENT APPLICATION 

(51) int Cl.^: H04L 12/56. H04L 29/06 



(84) Designated Contracting States: 


(72) Inventors: 


AT BE OH DE DK ES R FR GB GR IE IT U LU MC 


• Duffield, Nicholas G. 


NLPTSE 


Hoboken, New Jersey 07030 (US) 


Designated Extension States: 


• Stilladis, Dimitrios 


ALLTLVMKROSI 


Middletown, Monmouth, NJ 07748 (US) 




• Lalcshman, Tirunellal Viswanathan 


(30) Priority: 07.02.1997 US 37844 P 


Eatontown, IMonmouth, New Jersey 07724 (US) 


18.11.1997 US 




(71) Applicant: 


(74) Representative: 


Buckley, Christopher Simon Thirsk et al 


LUCENT TECHNOLOGIES INC. 


Lucent Technologies (UK) Ltd, 


Murray Hill, New Jersey 07974-0636 (US) 


5 Momlngton Road 




Woodlbrd Green, Essex IG8 OTU (GB) 



CM 
< 
CM 

to 

CO 

o 

Q. 

LU 



(54) Fair queuing system with adaptive bandwidth redistritxition 



(57) Apparatus for routing packets in a communica- 
tion network comprises a plurality (100) of per-connec- 
tion queues (20a-20i), each queue established for 
receiving packets from a respective source and tempo- 
rarily storing received packets before routing to a partic- 
ular destination; a weighted fair-queuing scheduler (40) 
for servicing packets from each of the plurality of per- 
connection queues at guaranteed pre-allocated rates; a 
sensing device for sensing a presence or absence of 
packets in queues, the ak)sence of packets in queues 



indicating availability of excess fc>andwidth; and. a state 
dependent scheduler (200) for redistributing excess 
bandwidth upon sensing of queues absent packets* the 
state dependent scheduler (200) servk:tng those 
queues in accordance with a state variat)le correspond- 
ing to a performance property of the queues, wherein 
delay and isolation properties for routing packets of 
respective queues In weighted feir-queuing is pre- 
served. 
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stream, and, the provision of isolation between streams, 
e.g., a misbehaving source will not effect the flow of 
other streams. Additionally, an added benefit is that 
when there is underutilization of capacity, e.g., when 
flow is particularly bursty and there may be Idle time, the 
WFQ system facilitates the redistribution of the unused 
bandwidth so as to preserve work-conservation prop- 
erty. Presently, the redistribution property of unused 
bandwidth capacity among the queues is done in a 
manner inherited from the fluid-flow nfxxlel, e.g., in 
accordance with the weight associated with the particu- 
lar queue. Thus, when the packet queues are idle, 
'excess' bandwidth is re^iistributed to bacMogged con- 
nections in proportion to their weights which are based 
on long-term requirements. 

The drawback of GPS that all fair queuing systems 
inherit in their dose emulation of GPS is that GPS 
severely restricts state-dependent bandwidtti sharing. 
The only state-dependency in GPS is in the number of 
bacMogged connections. There is no further latitude 
and sharing is determined by the guaranteed rates 
which are set based on long term needs of the connec- 
tbns. This restriction on bandwidtti sharing is more 
sti-ingent tiian that necessary to preserve a key of prop- 
erty of fair queuing, the ability to guarantee worst case 
delay bounds for leal^ bucket conti-olled traffic sources. 
Consequentiy, there is no need for fair queuing systems 
to emulate ttie possibly suboptimal excess bandwkJth 
sharing of GPS. 

It would thus be highly desirable to provide in a 
weighted fair queuing system emulating GPS, a method 
of achieving redistribution of unused t>andwkjth in a 
state-dependent manner, i.e.. tiiat reflects instantane- 
ous needs of the remaining bacMogged traffic f k>ws. 

Summary of Ihe Invention 

The instant invention is a modified approach to 
weighted fair queuing implementing an adaptive redistri- 
txjtion scheme. In such a scheme, each per connection 
fksw is guaranteed its specified share of the link band- 
widtti witti any excess bandwidtti being adaptively redis- 
tributed. The scheme enables preservation of fair 
queuing's ability to provide worst case end to end delay 
bounds and the schemes work like fair queuing when 
there is no excess bandwidth. The excess l^andwkfth 
may be distributed according to a different aiteria. 
Examples of state dependent criteria are: 1) Longest 
delay first (LDF) that serves the f kTw witti current long- 
est delay; 2) Least time to overflow (LTO) ttiat serves ttie 
flow with minimum difference between maximum 
allowed delay and current delay; 3) Least time to over- 
fbw witti leaky buctets (LTO-LB) ttiat senres ttie flow 
which would cause buffer overflow first if worst case 
arrivals happen. 

Advantageously, the adaptive bandwidth redistrHxi- 
tbn scheme always provides the minimum guarantees 
required for each of the connections and satisfies worst 



case fairness measures. 

The LDF policy uses excess bandwidtti to reduce 
ttie variance of the delay distribution which has the 
added benefit of reducing the playout txiffer size for 
5 voice and video sources. Simulations with video traces 
and with voice traffic shows that indeed this policy per- 
foons better than GPS wittiout any sacrifk:e of worst 
case guarantees. Since the deviation from the maxi- 
mum allowed delay is not taken into account, flows with 
10 small delay bounds (like voice) get alvnosX no excess 
bandwidth in the presence of flows with large delay 
bounds. Impredseness in assigning weights can result 
in these fkwvs experiencing losses much more than 
flows with large delay bounds. 
IS The LTO policy tries to minimize packet losses by 
assigning excess bandwkftti under the assumption that 
the fk}w which is likely to overfksw the quickest tias the 
most instantaneous bandwidtti need. In doing sa it 
takes into account the current deviatron of each flow 
20 from its maximum allowed delay. Simulations with a mix 
of CBR, voice, and video sources with very different 
delay bounds show that this policy reduces kisses for all 
classes as well as reduces the variance off delay for 
each class. The various features of novelty which char- 
ts acterize the invention are pointed out 

with particularity in the claims annexed to and form- 
ing a part of the disclosure. For a better understanding 
of the the invention, its operating advantages, and spe- 
cific objects attained by its use. reference shouM t>e had 
30 to the drawings and descriptive matter in which ttiere 
are illustrated and described preferred embodiments of 
the invention. 

Prief Pescriptipn of Prawings 

35 

FIG. 1 is a general data flow diagram illustrating ttie 
features of the weighted fair queuing and a rate propor- 
tional Server. 

FIG. 2 illustrates tiie weighted fair queuing schedul- 
40 ing system of the invention. 

Detailed Description of the Invention 

Figure 2 is a general biodk diagram illustrating the 
45 fair queueing method with adaptive excess bandwidtti 
redistribution. The system consists of a set of queues 
100, a shaper device 30a...30i associated with each 
queue, and Weighted Fair Queueing Sender or Rate 
Proportional Server 40 and a State Dependent Server 
50 (SDS) 200. Packets are temporarily stored in ttie 
queues 20a,...20i. A weight is assigned with each of 
ttiese queues representing the portion of the output 
bandwkfth ttiat must be allocated to the packets arriving 
in each of the queues. The shaper device is forwarding 
55 packets from the queues to the Weighted Fair Queueing 
Server with a rate exactiy equal to the allocated. This 
scenario enables excess bandwidth to become availa- 
ble for redistrSxition. In accordance witti the principles 
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of the invention, traffic from another source, i.e., another 
queue, may be directed to the adaptive k)andwidth redis- 
trilxition mechanism, which is hereafter refenred to as 
state dependent scheduler 200. ft should be understood 
to skilled artisans that the hardware architectures for 5 
implementation of the per-oonnection queues, shaper 
devices, and schedulers may vary, without diverting or 
straying from the principles of the invention described 
herein. 

As described above, the shaper releases packets to 
the scheduler vvith a rate exactly equal to the allocated 
rate. The service offered by the shapers 30a,...30i to the 
respective queue connections T during an interval of 
time is denoted by SjCx.t) in Rgure 2. It is assumed that 
packets are transferred from the shaper to the sched- 
uler with infinite capacity. The sendee offered t)y the rate 
proportional scheduler 40 is denoted as R\ (x.t). 

Packets that have not t)ecome eligible for service 
remain in the corresponding connection queue in the 
shaper, while ail the eligible packets wait for servk;e in 
the RPS Scheduler queue 40. Service is always pro- 
vided from the RPS scheduler queues 40. as long as 
packets are available there. When all of the scheduler 
queues are errpty, the state SDS 200 is invoked to 
select a packet from the queues 20a...,20i for transmis- 
sion with the service offered to a connection not affect- 
ing the state of the shaper 30. As shown in Rg. 2, the 
service offered by the shaper 30 to the queue connec- 
tion during an interval of time is denoted t)y Dj(t.t) 
and the servkse offered by the state dependent sched- 
uler 200 is denoted as Zj (x.t). 

The shaper device 30 can be implemented using a 
calendar queue that is a well known mechanism to any 
skilled artisans. A description of the calendar queue 
mechanism for shaping has been presented in D. Stili- 
adis and A. Verma: A General Methodology for Design- 
ing Scheduling and Shaping Algorithns, in Proceedings 
of IEEE INFOCOM 

The Rate Proportional Server (Weighted Pair 
Queueing Server) 40 can be implemented using any 
known fair queueing mechanisms of which there are 
several known to those skilled in the art. Examples are 
D. Stiliadis and A. Varnna: Traffic Scheduling System 
and Method for Packet-Switched Networks, U.S. Patent 
Application, Serial No. 08/634,904, filed April 15.1996. 

At some point of time, if it is determined that tfie are 
no packets in the RPS scheduler, then this is an indica- 
tion of free bandwidth. Thus, packets m^ be served by 
the SDS 200 based on a state dependent manner. By 
slate dependent manner, it is meant that the decision is 
t>ased on the current state of some variables related to 
the system. Two examples of state dependent decisfons 
are Longest Delay Rrst and Least Time to Overflow. 

In the Longest Delay First, the scheduler SDS 200 
selects anxHig the queues 20a,...20i. the one that has 55 
the packet that will see the longest delay if this queue is 
served with a rate equal to guaranteed. To do that, the 
scheduler assigns to each queue a delay value that is 



equal to the size of the queue divided by the guaranteed 
rate. This value describes the delay that the last packet 
of this queue will see. The SDS 200 selects the queue 
with the maximum such value using any mechanism for 
selecting the maximum value among a set of numbers. 
The selectfon mechanism is a well known method to 
any skilled artisan. 

In the Least Time to Overflow mechanism, the SDS 
scheduler 200 selects among the queues 20a,....20i 
that queue which is likely to overflow in the shortest 
time. The mechanism assumes that a maxinum size is 
associated with a queue. Every time a packet arrives or 
is serviced by a queue, a packet-to-overfk>w variable 
counts the number of packets that can k>e added to the 
queue without exceeding its maxinrum capacity. A sec- 
ond variable, denoted as time-to-overflow, indk^ates the 
expected time to overflow and is calculated by dividing 
the packet-tooverflow variat)le by the associated rate. 
The SDS 200 selects for transmission the queue with 
the minimum time-to-overffow variat)le. The minimum is 
determined by any of the well known mechanisms to 
anyone skilled in the art. Although the above variables 
explain two methods for cateulating state 

dependent variables, other similar methods may be 
used. Based on the state dependent variables, the 
scheduler mechanism 200 will direct packets from the 
selected queues over separate link 75 where the packet 
will be multiplexed by multiplexer element 350 and for- 
warded to its next destination. 

It should be understood that the property of worst 
case fairness is also satisfied in the system of the inven- 
tion. In worst case fairness, the servicing of queues of 
different lengths at different allocated rates is inter- 
leaved such that a packet from a first queue is serviced 
at time ti arxJ the packet of a second queue is serviced 
at a tnne X2. In the invention, the measure of worst ease 
feirness is satisfied, such that, the time interval \2'U* ^ 
the worst case, is less than or bounded by a value that 
is not a function of the numl>er of connections, but is a 
function of the packet size of the tongest queue. e.g.. 
the queue servfoed at time ti and the allocated rate of 
tfiat longest queue. 

Thus, the property of worst case faimess is satis- 
fied by the irrplementation of the SDS 200 in the 
weighted fair queuing system of the invention. 

The foregoing merely illustrates the principles of the 
present Invention. Those skilled in the art will be able to 
devise various modifications, which although not explic- 
itly described or shown herein, embody the principles of 
the invention and are thus within its spirit and scope. 

Claims 

1. Apparatus for routing packets in a communicalfon 
network comprising: 

a plurality of per-connection queues, each 
queue established for receiving packets from a 
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respective source and temporarily storing 
received packets before routing to a particular 
destination: 

weighted fair queuing scheduling means serv- 
icing packets from each of said plurality of per- 
connection queues at guaranteed pre-alh>- 
cated rates; 

means for sensing a presence or at)sence of 
packets in queues, said absence of packets in 
queues indicating availability of excess band- 
width; and. 

state dependent scheduling means for redis- 
tributing excess bandwidth upon sensing of 
queues at)sent said packets, said state 
dependent scheduling means servicing those 
queues in accordance with a state variable cor- 
responding to a performance property of said 
queues. 

wherein delay and isolation properties 
for routing packets of respective queues is pre- 
sented. 

2. Apparatus for routing packets in a communication 
network as claimed in Claim 1 , wherein a said per- 
formance property of a queue indudes the queue 
having the greatest amount of packets, said corre- 
sponding state variable being longest queue first. 

3. Apparatus for routing packets in a communication 
network as claimed in Claim 1. wherein a said per- 
formance property of a queue includes that queue 
having a packet waiting to be serviced in the long- 
est amount of time, said corresponding state varia- 
ble being tongest delay first. 

4. Apparatus for routing packets in a communication 
network as claimed in Claim 1, wherein a said per- 
formance property of a queue indudes the queue 
most likely to overflow buffer memory containing 
said queue, sakj corresponding state variable being 
most likely to cverf bw buffer. 

5. Apparatus for routing packets in a communication 
network as claimed in Claim 1, wherein a worst 
case fairness measure for said connections is satis- 
fied. 

6. Method for routing packets in a communication net* 
work comprising: 

establishing a pluiality of per-connection 
queues, each queue for receiving packets from 
a respective source and temporarily storing 
received packets t>efore routing to a particular 
destination; 

senfidng pactets from each of said plurality of 
per-oonnection queues at guaranteed pre-allo- 
cated rates; 



sensing a presence or absence of packets in 
queues, sakJ absence of packets in queues 
incficating availability of excess bandwidth; and. 
for redistrbuting excess t>andwidth upon sens- 

5 ing of queues absent said packets, said state 

dependent scheduling means servidng those 
queues in accordance with a state variable cor- 
responding to a performance property of sakJ 
queues, white preserving delay and isolation 

10 properties for routing of said packets. 
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